Wrap dask_setup scripts in if __name__ == "__main__" guard; use threaded scheduler for import scripts#127
Merged
akhanf merged 10 commits intoperf-updatesfrom Mar 10, 2026
Conversation
…scheduler for ome_zarr_to_nii, create_imaris_crops, create_patches Co-authored-by: akhanf <11492701+akhanf@users.noreply.github.com>
Copilot
AI
changed the title
[WIP] Update Python scripts to use main block and threaded execution
Wrap dask_setup scripts in Mar 10, 2026
if __name__ == "__main__" guard; use threaded scheduler for import scripts
Refactored code to use Dask client for parallel processing and improved structure.
akhanf
added a commit
that referenced
this pull request
Mar 14, 2026
* localrule and cpus for vesselfm * more fixes * remove job grouping, update perf * add group_jobs to temp() outputs with work * update group_jobs and resources * fix * Wrap dask_setup scripts in `if __name__ == "__main__"` guard; use threaded scheduler for import scripts (#127) * Wrap dask_setup scripts in if __name__ == "__main__" and force threads scheduler for certain jobs * fixes * update runtimes * retain .ozx (zip ome zarr) instead of only temp ome.zarr still written to tmp first, then zipped to shared fs. hopefully still is fast for reading, but ultimately ideal for retaining outputs * up short runtimes * use otsu only * reduce memory * don't save upsampled bias field * update resources, chunks, and no patches by default * another resource fitx * update resources * adjust resources * fix resources * bugfix for zarrnii units, was incorrectly changing units related to zarrnii issue #203, fix here in any case * fmt --------- Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When using
dask.distributed, worker processes are forked and re-import the script, causing top-level code to execute unintentionally. All scripts usingget_dask_clientmust gate execution behindif __name__ == "__main__":.Changes
All 14
dask_setupscripts — moved all code (including any pre-with get_dask_client(...)setup that accessessnakemake) insideif __name__ == "__main__":3 scripts hardcoded to
"threads"scheduler —ome_zarr_to_nii.py,create_imaris_crops.py,create_patches.pynow bypasssnakemake.config["dask_scheduler"]and always use the threaded scheduler:✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.